[DEV-12163] - Office Loader - #14
Conversation
| empty_pull_count += 1 | ||
| continue | ||
| entries_processed = 0 | ||
| with tqdm(total=total_expected_records, desc=f"Level {level}", file=sys.stdout) as pbar: |
There was a problem hiding this comment.
This tqdm library is optional but adds a cool progress bar and timing stats to any iterable process, which is great for monitoring and debugging.
There was a problem hiding this comment.
Nice, curious how this will translate in Jenkins/NPC but I'm fine with it if it turns out well.
dpb-bah
left a comment
dpb-bah
left a comment
There was a problem hiding this comment.
Excellent work translating the original script to this from an initial scan.
Following the bronze/silver/gold paradigm with the shared Data Model.xlsx, can you also include the OfficeBronze model that'll just include the raw results from the API, and then OfficeGold is based on that?
| ) | ||
|
|
||
|
|
||
| class OfficeGold(CSVModel): |
There was a problem hiding this comment.
Can you add this to brus_backend_common/models/__init__.py so it'll be picked up in LAKEHOUSE_MODELS and brus_backend_common/scripts/create_migrate_lakehouse_model.py?
| DB1_URL: SecretStr = SecretStr("") | ||
| DB2_URL: SecretStr = SecretStr("") | ||
|
|
||
| # sam.gov |
There was a problem hiding this comment.
Good additions. Can you
- include them in the .env.template?
- and put some dummy data in the test .conf (
.github/.env.test)?
| JAVA_VERSION: str = "1.8.0" | ||
| SPARK_VERSION: str = "3.4.1" | ||
| HADOOP_VERSION: str = "3.3.4" | ||
| SCALA_VERSION: str = "2.12" | ||
| DELTA_VERSION: str = "2.4.0" |
There was a problem hiding this comment.
I'm fine with setting these here to set what's expected (matches .github/.env.test). If we're setting defaults, could you also set them in .env.template?
| DATABASE_NAME = LakeHouseDatabase.BRONZE | ||
| TABLE_NAME = "office" | ||
| DESCRIPTION = "Raw office data" | ||
| CSV_NAME = "office.csv" |
There was a problem hiding this comment.
To reduce any confusion with both files being named office.csv can you name this one raw_office.csv or the gold version office_gold.csv? (I see I did the same with FON so I'll be updating those too).
zachflanders-frb
changed the title
2 participants
Description:
This PR port the old load_federal_hierarchy script into our new pattern. I made a number of improvements to the pandas code to vectorize transformations that occured row or item-wise in the old code. I also added more asynchronous code to assist in the getting of office records.
Requirements for PR Merge:
Explain N/A in above checklist: